We have tests that run on an 8GB RAM instance with no problems but when downgrading to 4GB, the tests abruptly stop and exit with code 137. I ssh'd into the instance and followed the output of top but the %MEM used does not go above 5%.
The tests run fine locally using no where near even 1GB RAM as well. Could the number of vCPUs be an issue?
https://circleci.com/docs/2.0/configuration-reference/#resource_class
~$ dmesg | grep -i oom
[97139.572780] node invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
[97139.572824] [<ffffffff8118a335>] oom_kill_process+0x205/0x3d0
[97139.572833] [<ffffffff811f431d>] mem_cgroup_oom_synchronize+0x30d/0x330
[97139.573080] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[107600.844934] postgres invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
[107600.844978] [<ffffffff8118a335>] oom_kill_process+0x205/0x3d0
[107600.844984] [<ffffffff811f431d>] mem_cgroup_oom_synchronize+0x30d/0x330
[107600.845178] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Someone on ##linux recommended this command which I'm not sure how to parse.
Otherwise what are the steps one should take to debug this?
If this helps this is a scala play app with these options.
SBT_OPTS: "-Xmx2G -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Xss2M -Duser.timezone=GMT"
JAVA_OPTS: "-Xmx2G -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Xss2M -Duser.timezone=GMT"